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EXAMINING FEEDBACK IN AN INSTRUCTIONAL VIDEO GAME USING PROCESS 


DATA AND ERROR ANALYSIS 

Rebecca E. Buschang, Deirdre Kerr, Gregory K.W.K. Chung 
CRESST/University of California, Los Angeles 


Abstract 

Appropriately designed technology-based learning environments such as video games can be 
used to give immediate and individualized feedback to students. However, little is known 
about the design and use of feedback in instructional video games. This study investigated 
how feedback used in a mathematics video game about fractions impacted student actions in 
the game. Results indicated the type of feedback did not significantly affect student actions. 
Process data were also analyzed to identify specific student errors as well as opportunities to 
provide feedback for future versions of the game. Results of this study suggest that process 
data are a unique feature of technology-based learning environments that can be used to 
analyze errors and create targeted feedback for students. 

Introduction 


Objective 

The purpose of this exploratory study was to examine the impact of feedback on student 
success in a video game for learning, and to determine how process data can be used to enhance 
feedback. 

Theoretical Framework 

Feedback is information given to students with the goal of improving learning through 
changing a student behavior or how a student thinks about a problem. A recent review of the 
literature suggests that effective feedback must include both verification feedback (i.e., indicates 
whether the answer is correct or incorrect) and elaboration feedback (i.e., addresses specifics 
about student performance, such as particular errors students made, gives guidance, or provides 
examples) (Shute, 2008). 

In a classroom, feedback given by the teacher requires the teacher to evaluate and analyze 
individual student artifacts. The analysis of each student’s work requires valuable time that 
teachers may not be able to spare. In addition, there is evidence that teachers have difficulty 
evaluating student artifacts to determine instructional steps (Heritage, Kim, Vendlinski, & 
Herman, 2008; Herman, Osmundson, & Silver, 2010). However, technology-based tools, if 
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designed appropriately, can provide a unique environment where feedback and instruction can 
occur automatically and can be targeted to student misunderstandings or common errors. 

Instructional video games are one type of technology in which feedback and instruction can 
be naturally incorporated. Level success feedback is inherent to video games. Students 
automatically know if they successfully solved the level because they either move forward in the 
game or must re-attempt the level. Additionally, elaborative feedback can be included in the 
game to explain why students succeeded or failed at the level and give guidance on the correct 
solution. Nonetheless, the research literature on feedback in instructional video games is limited, 
and process data are rarely used to analyze student errors and determine appropriate feedback. 
Therefore, this study explores the following research questions: 

1. Does type of feedback impact student actions in the video game? 

2. How can process data be used to improve feedback? 

Method 

One hundred and eighty-seven middle school students were asked to play a mathematics 
video game aimed at teaching key topics of rational numbers including identification of unit size, 
the numerator, the denominator, and addition of fractions (Chung, 2010). In each level of the 
game, students were asked to determine the fraction needed to jump a certain distance from the 
pink block to the blue “X” block (see Figure 1). 



Figure 1. Screen shot of example game level. 

Students were randomly assigned to receive either (a) both level success feedback and 
individualized elaborative feedback or (b) only level success feedback. Level success feedback 
was given in both conditions at the end of each level, and verified if the solution was correct or 
incorrect. If a solution was correct, the student was passed to the next level. If a solution was 
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incorrect, the character died at the point of the mistake. Individualized elaborative feedback was 
triggered by specific actions taken by students and resulted in feedback that addressed specific 
mistakes students were making (e.g., “The fraction denominators don’t match!”), highlighted 
certain features to pay attention to (e.g., “Did you notice ...”), and provided guidance to 
successfully complete that particular level (e.g., “Remember: % + % = 1/8”). 

Data Sources 

Process data for three consecutive levels were collected as students played the game and 
logged student actions. This type of data can be used to determine the types, frequencies, and 
timing of student errors (Romero & Ventura, 2007). Process data in this study were used to (1) 
identify the types of errors and solutions for each analyzed level and (2) determine how many 
times students attempted a level. Types of errors were identified through cluster analysis (Kerr, 
Chung, & Iseli, 2010). 

To answer research question 1 and determine if there were significant differences between 
conditions for the number of attempts students made or the types of errors students made on that 
level, an independent t test was conducted. To answer research question 2 and determine how 
process data could be used to improve feedback, the types of errors and number of students 
making each type of error by attempt were analyzed and graphed. For similar levels, the types of 
errors were compared. 

Results and Conclusions 

Research question 1: Does the type of feedback impact student actions in the videogame? 

Results of the independent samples t test indicated that on Levels A and B, no significant 
differences were found between conditions on the number of attempts or the number of specific 
errors made in each level. On Level C, those who received both types of feedback attempted the 
level significantly fewer times (M = 3.39, SD = 2.37) than those who received only level success 
feedback (M = 3.93, SD = 2.87), t(594) = 2.50, p = .02, d = .21. No differences were found 
between conditions on the types of errors made. 

Research question 2: How can process data be used to improve feedback? 

The results indicate that of the 156 students who attempted Level A (see Figure 2), only 
about 6% solved the level on the first attempt (see Table 1). Error analysis of the first attempt 
indicates that approximately 30% of students attempted to solve the level using a mixed number, 
approximately 15% were using whole units only, and approximately 3% were ignoring the unit 
bars and using a denominator of thirds. An additional 46% of errors made in the first attempt 
were unidentifiable. Results also indicate that the number of errors drops off over time, and after 
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approximately the fourth attempt, most students have successfully completed the level (see 
Figure 3). 


Ignore End of Unit 

4 >-*— •— a 

Seeing as a Mixed Number: 


Figure 2. Screen shot of Level A and error pattern explanations. 




Table 1 


Percent Solutions and Errors for the First Attempt of Each Level 


Type of solution or error 


First attempt at solving level 



Level A 

Level B 

Level C 


(n = 156) 

(n = 146) 

(n = 138) 

Correct solution 

6% 

15% 

18% 

Ignoring unit bars 

3% 



Using whole units only 

15% 

23% 

4% 

Using alternative form of correct 
solution 

30% 

— 

— 

Smaller denominator than needed 

— 

50% 

25% 

Larger denominator than needed 

— 

— 

4% 

Reset correct solution 

— 

— 

7% 

Unknown errors 

46% 

14% 

42% 
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Level A: Number of Error Types by Attempt 


Types of Errors 

- Ignored end of unit 
--- Jumped whole units only 

Used alternative form of 
correct solution 


1.00 2.00 3.00 4.00 5.00 6.00 7.00 

attempt 

Figure 3. Number of students making errors by attempt. 

In this game, improper fractions must be used instead of mixed numbers to solve the level. 
Therefore, results indicate that these students knew the answer to the level, but that most were 
unable to use the appropriate form until after several attempts. Feedback to these students should 
highlight alternative forms of correct solutions. These results also indicate that only a few 
students ignored the whole unit bar which is an error students typically make. In addition, many 
students only used whole numbers to solve the level. This indicates a significant 
misunderstanding at this level of the game, and reveals that major feedback relating to 
identification of the units, numerators, and denominators is necessary for these students. Since 
approximately half of the errors were unidentifiable into any one error or solution type, many 
errors were not evaluated in this analysis. Further work must be completed to identify these 
errors. 

Results of the analysis of errors for Level B (see Figure 4) indicate that of the 146 students 
who attempted this level, approximately 15% solved it correctly on the first attempt (see Table 
1). Error analysis of the first attempt indicated that approximately 50% used a denominator of 
halves, an additional 23% used only whole units, and 14% of the errors were unidentifiable. 
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Results also indicate that the number of errors drops off over time, and after approximately the 
fourth attempt, most students have successfully completed the level (see Figure 5). 



j ump 


refur tv** 
to 4^; 

tutorial 


trampolines 


lhi«iip 










Partitioning Exclusively: 



Using a v- 

Denominator of 
Halves 


Figure 4. Screen shot of Level B and error pattern explanation. 


Level B: Number of Error Types by Attempt 



attempt 


Figure 5. Number of students making errors by attempt. 
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These results indicate that many students made a common error and incorrectly identified 
the denominator as halves. Feedback for this common math error should highlight where the unit 
bar starts and ends, and how to determine the denominator. A higher percentage of students on 
Level B used only whole numbers to solve the problem than on Level A. This indicates that the 
feedback given in Level A was not sufficient to remediate this error. This evidence also indicates 
these students did not fully understand how to identify the unit and the denominator. The 
feedback to these students should be substantial. Fewer errors in this level were unidentifiable as 
compared to Level A. 

Level C (see Figure 6) was very similar to Level B. Therefore, results between the two 
levels are compared in this analysis. Results of the analysis of errors for Level C indicate that of 
the 138 students who attempted this level, approximately 18% solved it correctly on the first 
attempt (see Table 1). Error analysis of the first attempt indicated that approximately 25% used 
thirds as the denominator, 4% used fifths as the denominator, 4% used whole units exclusively, 
and 7% reset a correct solution before completing the level during the first attempt. Forty-two 
percent of the errors were unidentifiable. Results also indicate that the number of errors drops 
off, and after approximately the fourth attempt, most students have successfully completed the 
level (see Figure 7) 

These results indicate that half the number of students as on the previous level were 
incorrectly identifying the denominator as smaller than necessary, and provide evidence that 
students may be learning from Level B to Level C. Additionally, fewer than 5% as compared to 
23% in Level B are using whole units only. This provides additional evidence that students may 
be learning from one level to the next. Using a larger denominator than needed emerged as a new 
error in this level. However, only 4% of students completed this action indicating that they were 
not identifying the denominator correctly. These students need feedback that helps them 
determine how to correctly identify the unit and denominator. 
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Figure 6. Screen shot of Level C. 


Level C: Number of Error Types by Attempt 



attempt 


Figure 7. Number of students making errors by attempt. 
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Scholarly Significance 

First, this study analyzed the impact of different types of feedback on student performance. 
Results indicated that in most cases, the combination of elaborative and level success feedback 
as initially designed did not lead to significantly different student actions than only providing 
level success feedback. Because no differences were found between conditions, process data 
were used to analyze error rates and determine if feedback should be modified. Analyzing 
process data gave insight into the types of errors being committed and the number of attempts 
being made at each level. This information can be used to create more targeted feedback based 
on the data. This analysis also indicates that process data should be explored as an indicator of 
learning from level to level. Research such as this is important because it highlights how process 
data can be used to determine the types of errors made by students, and how feedback can be 
improved to target student understanding. Further analysis of all levels of the game is necessary 
to determine the full effect of the feedback on the number of attempts needed to solve each level 
and on the types of errors being made in each level. 
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